1. Field of the Invention
The present invention relates to an image processing method, image processing apparatus, and program. Particularly, the present invention is suitably applied when handling a plurality of image data, correction data, and the like in a main memory for image processing with dividing them, and sequentially reading out necessary data to perform image processing and correction processing.
2. Description of the Related Art
Conventionally, local or neighbor image processing such as spatial filtering is done when forming an image. Local image processing is image processing which executes any calculation using all pixels falling within a spatial filtering region including a pixel to be processed. In the following description, a pixel to be processed will be simply referred to as a pixel of interest.
For example, spatial filtering such as edge enhancement processing or blur processing is performed for digital image data 300 in FIG. 11. As a conventional technique for the local image processing, digital image data is divided into areas and local image processing is executed in a unit of area created by division (see U.S. Pat. No. 6,587,158, Japanese Patent Laid-Open No. 2000-312327, Japanese Patent No. 3733826, and Japanese Patent Laid-Open No. 2006-139606).
According to this technique, one entire digital image data 300 is generally divided in a band-like or strip shape, as shown in 11a to 11d of FIG. 11. The respective areas sequentially undergo various kinds of image processes. The elongated area created by division is called a band area. A storage area where the band area is rendered is called a band memory. Processing of dividing an image is called band division. The band memory is not always reserved as a storage area in the main memory and may be reserved in any storage area in the system. In the following description, however, the band memory is reserved in the main memory for descriptive convenience.
As the coordinate system (main scanning direction−sub-scanning direction) of digital image data, a new coordinate system (band area coordinate system) based on the directions of length and height is defined and expresses the band area by (length Bdl×height Bdh). When dividing an image into bands in the sub-scanning direction, the band area is represented by (length Bdl×height Bdh), as shown in 11e of FIG. 11. When dividing an image into bands in the main scanning direction, the band area is represented by (length Bdl×height Bdh), as shown in 12a to 12d of FIG. 12.
More specifically, the length Bdl of the band area takes the value of either the width of digital image data in the main scanning direction or the height in the sub-scanning direction. The band height takes an arbitrary value.
Band processing will be explained in more detail using the example of FIG. 11. This also basically applies to FIG. 12, and a repetitive description will be omitted.
A first band area 301 shown in 11a of FIG. 11 is rendered in the band memory in the main memory and undergoes image processing. Then, a second band area 302 shown in 11b of FIG. 11 is rendered and overwritten in the band memory where the first band area 301 has been rendered, and undergoes image processing. Further, a third band area 303 shown in 11c of FIG. 11 is rendered and overwritten in the band memory where the second band area 302 has been rendered, and undergoes image processing. Finally, a fourth band area 304 shown in 11d of FIG. 11 is rendered and overwritten in the band memory where the third band area 303 has been rendered, and undergoes image processing. As is apparent from 11a to 11d of FIG. 11, the band areas 301 to 304 have the same length, but their heights need not always been equal. The height of the band memory serving as a storage area reserved in the main memory is determined by a band area largest in size in the direction of height (first to third band areas 301 to 303 in the example of FIG. 11). The explanation of FIG. 11 also applies to the example of FIG. 12 except that the band division count is three.
According to this technique, respective band areas are designed to partially overlap each other at the boundaries between adjacent areas in order to perform local image processing without any gap between the band areas. In Japanese Patent Laid-Open No. 2006-139606, an additional pixel is scanned in the same direction as the height of each band area. The capacity of a delay memory that holds a pixel of interest necessary for local image processing is defined by the height of each band area, thereby reducing the capacity of the delay memory.
In the field of image processing methods and apparatuses, however, there are many image processes which cannot be dealt with by considering only local or neighbor image processing such as spatial filtering described in U.S. Pat. No. 6,587,158, Japanese Patent Laid-Open No. 2000-312327, Japanese Patent No. 3733826, and Japanese Patent Laid-Open No. 2006-139606. Such image processing will be exemplified with reference to FIGS. 13A to 13C.
An example is correction processing 1010 to correct the individual differences between the reading elements of a sensor device for an image read by an image reading apparatus such as a scanner, as shown in FIG. 13A. Pieces of information necessary in the correction processing 1010 are, for example, minimum read value data and maximum read value data for each reading element of the sensor device. Based on these pieces of information, a target read image is corrected by calculation using equation (1) for each pixel data (pixel value):X=(P−MIN)/(MAX−MIN)×1023  (1)where X is a corrected pixel value (X for a 10-bit digital signal), P is a read pixel value, MIN is a minimum read value, and MAX is a maximum read value.
That is, data necessary in the correction processing 1010 are three different types of information, i.e., minimum value data, maximum value data, and pixel data of a read image that change for each pixel. When the sensor device of the scanner is a line sensor having a line of reading elements corresponding to respective pixels, the sensor moves in the sub-scanning direction to read a two-dimensional image. In this reading method, minimum value data and maximum value data, which change for each pixel, are aligned in the main scanning direction. In the sub-scanning direction, the same minimum value data and maximum value data are aligned for pixels at the same main scanning position.
Correction processing similar to the above-described one is one for a two-dimensional display device used in a flat-screen television such as a liquid crystal television or plasma television. In the two-dimensional display device, the individual differences between two-dimensionally spread display elements change. Thus, correction processing is executed using two-dimensional correction data, instead of one-dimensional correction data in the correction processing 1010. This correction processing needs to pay attention to the repetitive method of correction data because the repetitive method changes depending on the device shape (corresponding to a method of spreading elements).
Another example is combination processing 1020 to combine a plurality of rendering images in accordance with combination data (α value), as shown in FIG. 13B. When two rendering images are to be combined, the combination processing 1020 is done for each pixel using a combination equation like the following equation (2):X=P2×A+P1×(1024−A)  (2)where X is a combined pixel value (when the α value is given by a 10-bit digital signal), P1 is the pixel value of the first rendering image, P2 is that of the second rendering image, and A is the α value of each pixel of combined data. Note that the combined data (α value) may be equal for all the pixels of a rendering image, or may be two-dimensional combined data (α value) which changes for each pixel of a rendering image.
For the latter combined data (α value), when the number of rendering images to be combined is N, the numbers of pixel data necessary for processing and the number of combined data (α value) differ from those in the foregoing example in the combination processing 1020.
In some cases, adaptive spatial filtering is done by referring to a plurality of field images which are successive along the time axis, like interlaced/progressive conversion (to be simply referred to as I/P conversion 1030) of a moving image, as shown in FIG. 13C. Generally in image processing referring to a plurality of field images, like the I/P conversion 1030, high-quality image processing can be achieved by adaptive processing using a larger number of field images.
A plurality of field images are generally stored in an external memory such as a low-cost DRAM connected to a semiconductor chip (e.g., ASIC) on which an image processing circuit is mounted. In every I/P processing, therefore, a plurality of field images need to be read out from the external memory and input to the I/P conversion circuit. Owing to a limitation on product cost, most external memories have a limitation on the memory band permissible when reading out a plurality of field images from the external memory. It is impossible to read out field images having high resolution such as full high vision resolution (1920×1080i), and execute I/P conversion without any limit.
Assume that there is an image processing apparatus with a memory band capable of reading out, for example, up to three field images having the full high vision resolution. Apparently, when I/P-converting a moving image having low resolution like the NTSC resolution (720×480i), this image processing apparatus can achieve high-quality I/P conversion by referring to a larger number of field images than those of a moving image having the full high vision resolution.
In the I/P conversion 1030, it is important to flexibly switch the image processing method in accordance with the resolution and reference field count in order to perform image processing at as high quality as possible when the available memory band is constant.
It is common to the above-described image processes in FIGS. 13A to 13C that it is not sufficient to perform image processing in consideration of only one of two-dimensionally arranged image data. In some cases, correction data is necessary, combined data is necessary, or the number of image data for use is arbitrary. The method of acquiring image data, correction data, or combined data necessary for the image processing changes depending on the sensor shape. For example, data needs to be repetitively read out from a storage device such as an external memory. The number of rendering images to be read out may change depending on a number designated by the user. The number of field images to be read out may change according to the input resolution of the broadcast wave.
These days, the resolution of digital image data is increasing to improve the image quality for differentiation of devices. Along with this, the data processing amount in image processing is increasing. Implementation of higher-quality, advanced-function products require various kinds of information. However, in product development, it is desired to meet various image processing requests at limited cost. In other words, demand has arisen for an apparatus and method which flexibly perform a variety of image processes described above while keeping the memory band and circuit scale constant.
The conventional technique mentions capacity reduction of the delay memory for spatial filtering by reading out one image data from an external memory for each band area when performing local image processing. However, the conventional technique does not examine a flexible change of the band area in accordance with the resolution and number of data for a plurality of correction data for one image data, a plurality of image data, and a plurality of successive field images. The number of data is the number of reference fields, that of rendering images, or that of correction data.